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FORMULATION  OF  PROVIDER  R&D  OUTPUT  METRICS 
FOR  NAVY  WARFARE  ENTERPRISES 


1.  INTRODUCTION 


A  metric  is  a  standard  of  measurement.  According  to  Geisler  (reference  1 ),  a  metric  “is  a 
description  of  a  system  of  measurement  that  includes  the  item  being  measured,  the  unit  of 
measurement,  and  the  value  of  the  unit.”  In  his  monumental  work  on  science  and  technology 
metrics,  Kostoff  (reference  2)  provides  a  reminder  that  every  “metric  and  associated 
data... should  have  a  decision  focus;  it  should  contribute  to  the  answer  of  a  question  which  in 
turn  would  be  the  basis  of  a  recommendation  for  future  action.” 

This  leads  to  the  following  necessary  steps  in  constructing  metrics  (adapted  from  reference 

2): 


•  Determine  what  we  want  to  measure 

•  Determine  what  we  want  to  achieve  with  our  measurement 

•  Determine  available  measures 

•  As  necessary,  create  combinations  of  measures  such  as  indexes  and  integrated 
measures 

•  Select  methods  and  instruments  for  data  collection 

•  Assess  the  validity,  reliability,  and  amplitude  of  the  metric. 

So,  to  be  effective,  a  metric  must  have  units,  be  measurable,  address  a  particular  question, 
and  be  of  a  form  that  leads  to  informed  decision  making.  Without  this  construct,  a  metric 
becomes  meaningless. 

Implementation  of  the  Navy  Enterprise  construct  has  generated  considerable  discussion 
pertaining  to  output  metrics.  The  drive  toward  output  metrics  is  occurring  in  parallel  with  the 
evolution  of  the  Navy  Enterprise  construct.  Selection  of  output  metrics  has  been  particularly 
elusive  for  research,  development,  test,  and  evaluation  (RDT&E)  functions  of  the  Warfare 
Enterprises.  Research  and  development  (R&D)  value  metrics  have  traditionally  been  difficult  to 
generate  and  measure  within  the  public  and  private  sector  technical  communities. 

This  report  draws  on  R&D  metrics  research  as  applied  to  the  private  and  public  sectors. 

Care  is  taken  to  investigate  whether  private  sector  experience  is  applicable  to  the  Navy 
Enterprises.  An  attempt  is  made  to  go  back  to  first  principles  and  adhere  to  business 
management  terminology.  As  such,  it  is  not  an  entering  assumption  that  the  measures  of  Navy 
Enterprise  or  Navy  Warfare  Center  effects  (i.e.,  value  to  the  warfighter)  will  be  determinable 
with  a  single  overarching  output  metric. 
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This  report  begins  with  a  discussion  of  Navy  Enterprises  and  a  perspective  on  Warfare 
Enterprise  output  metrics  based  on  the  warfighting  value  of  the  provider  product  and  services. 
The  second  section  provides  a  discussion  of  R&D  metrics  as  applicable  to  Warfare  Enterprises 
and  concludes  with  a  discussion  of  provider  R&D  organization  output  metrics.  The  report  does 
not  address  Navy  Enterprise  metrics  associated  with  acquisition  or  industrial  operations. 
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2.  NAVY  ENTERPRISES 


2.1  ENTERPRISE  CONSTRUCT 

The  Navy’s  alignment  to  an  enterprise  construct  can  be  traced  to  Chief  of  Naval  Operations 
(CNO)  Admiral  Vem  Clark’s  proclamation  of  Sea  Power  21  as  the  vision  of  how  the  Navy 
would  "organize,  integrate,  and  transform”  (reference  3).  Sea  Enterprise  was  introduced  within 
Sea  Power  21  as  a  supporting  organizational  process  with  the  express  purpose  of  finding 
organizational  and  process  efficiencies  that  would  result  in  the  savings  necessary  to  recapitalize 
the  Navy  and  transform  it  "into  a  21s,-century  force  that  delivers  what  truly  matters:  increased 
combat  capability”  (reference  3).  The  CNO,  within  his  2003  leadership  guidance  (reference  4), 
tasked  OPNAV  N09/N4  with  the  establishment  of  “a  Sea  enterprise  organization  and  process 
that  coordinates  efforts  between  the  Navy  Secretariat,  USMC,  and  other  Services,  to  generate 
corporate  efficiencies  for  reinvestment.”  The  guidance  was  greatly  expanded  in  2004  (reference 
5)  to  include  the  application  of  the  Sea  Enterprise  principles  and  the  establishment  of,  among 
other  things,  various  cost-benefit  analyses,  savings  generation,  best  practice  sharing,  and  metric 
programs.*  The  business  of  Sea  Enterprise  has  been  progressing  since. 

The  Navy  Enterprises  are  based  on  a  construct  of  warfighter ,  provider,  and  resource  sponsor 
relationships  as  shown  in  figure  1 .  The  warfighter  is  responsible  for  establishing  the  integrated 
warfighting  requirements.  The  provider  is  responsible  for  the  processes  and  intellectual  capital 
that  allow  requirements  and  resources  to  be  transformed  into  readiness  and  capability  delivered 
to  the  Fleet.  The  resource  sponsor  is  responsible  for  providing  the  resources  necessary  to  support 
the  provider  (reference  6).  The  enterprise  construct  is  based  on  the  premise  that  all  three 
elements  must  be  synchronized  and  transparent  if  the  required  operational  capabilities  are  to  be 
delivered  in  the  most  efficient  manner  (in  terms  of  both  cost  and  schedule). 


Figure  /.  Navy  Enterprise  Construct 


*ln  reference  5,  the  staled  Sea  Enterprise  principles  are:  ( 1 )  Leverage  technology  to  improve  performance  and 
minimize  manpower  costs.  (2)  Promote  competition  and  reward  innovation  and  efficiency.  (3)  Challenge 
institutional  encumbrances  that  impede  creativity  and  boldness  in  innovation.  (4)  Aggressively  divest  non-core, 
under-performing,  or  unnecessary  products,  services,  and  production  capacity.  (5)  Merge  repetitive,  redundant,  or 
superfluous  costs.  (6)  Minimize  acquisition  and  life-cycle  costs.  (7)  Maximize  in-service  capital  equipment 
utilization.  (8)  Challenge  every  assumption,  cost,  and  requirement. 
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The  Navy  has  established  a  hierarchy  of  enterprises  where  each  tier  is  based  on  this 
construct.  Figure  2  shows  the  hierarchy  promulgated  by  Commander,  U.S.  Fleet  Forces 
Command  (USFFC)  in  2005  (reference  7).  The  warfighter,  resource  sponsor,  and  provider 
change  at  each  tier  in  concert  with  their  organizational  missions  and  responsibilities.  The 
warfighter  at  the  lower  level  enterprise — the  Warfare  Enterprise — becomes  the  provider  to  the 
next  higher  level.  In  this  manner,  systems  and  personnel  comprise  the  units  (platforms — hulls, 
airframes,  and  networks),  units  comprise  Fleets,  and  Fleets  comprise  the  highest-level  Navy 
response  for  the  nation. 


Enterprise  Construct 


« 

SECI 

CNO 

_  i 

Productivity  - 
Readiness  /  Cost 


Strategic  Guidance 

IMowt.  X  and  Fiscal 

Navy  X  Decisions 

Enterprise 


Ops /  Readiness  Direction 
V  and  Rqmts 
Fleet  X.  Prioritization 
Readiness 
Enterprise 


l=> 


FRP/FRTP> 

All  Forces 
Ready  for 
Tasking 

Units  Ready 

for  Tasking 

Figure  2.  Hierarchy  of  Navy  Enterprises 


The  Navy  has  instituted  a  number  of  Warfare  Enterprises.  Currently,  the  Warfare  Enterprise 
domains  are  (reference  8): 

•  Naval  Aviation  Enterprise  (NAE) 

•  Naval  Surface  Warfare  Enterprise  (SWE) 

•  Naval  Undersea  Enterprise  (USE) 

•  Naval  Expeditionary  Combat  Command  Enterprise  (NECC) 

•  Naval  NETWAR  FORCEnet  Enterprise  (NNFE). 
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As  stated  in  reference  8, 

The  primary  mission  of  (he  five  Warfare  Enterprises  is  to  deliver  warfare 
capabilities  in  response  to  the  Navy  Component  Commander's  and  Combatant 
Commander’s  demand  signals  while  working  to  enhance  effectiveness  and 
efficiency,  thereby  increasing  productivity  across  their  domain  and  driving  out 
cost. 

The  multi-tier  nature  of  the  Navy  Enterprise  construct  results  in  the  following  relationships 
among  the  elements,  as  shown  in  figure  3: 

•  The  upper  tier  informs  the  warfighter  in  the  lower  tier  of  its  “demand  signal'’ — i.e.,  the 
requirements  from  the  operating  forces. 

•  The  warfighter  informs  the  resource  sponsor  and  provider  of  these  needs. 

•  The  resource  sponsor  works  with  the  warfighter  and  provider  to  optimize  the  delivered 
products  and  services  within  funding  allocations. 

•  The  provider  provides  the  warfighter  with  the  products  and  services  as  tasked  and 
funded  by  the  resource  sponsor. 


Figure  3.  Enterprise  Participant  Roles 


The  strategic  objectives  of  Sea  Enterprise  are  to  change  culture  and  behaviors,  improve 
processes  and  structures,  and  harvest  savings  (reference  9).  These  objectives  are  to  be  met  by  the 
Warfare  Enterprise  processes  and  organizational  mission  and  responsibilities.  It  is  important  to 
understand  the  level  of  authority  associated  with  a  Navy  Enterprise  as  one  of  engagement  and 
process  improvement  through  collaboration,  coordination,  sharing,  transparency,  and  linked 
metrics.  It  is  not  an  alternate  process  for  line  management  command  and  control  responsibilities; 
therefore,  it  is  not  a  substitute  for  the  personal  authority  and  responsibility  to  decide,  command, 
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order,  or  comply.  Commander,  Naval  Sea  Systems  Command  (COMNAVSEA)  guidance 
relative  to  the  lower-tier  Warfare  Enterprise  summarizes  the  intent  as  follows  (reference  10): 

Enterprise  management  is  a  behavioral  model,  not  a  command  and  control 
structure.  The  model  facilitates  collaboration  with  all  readiness  partners  and 
contributing  organizations  through  a  structured  approach,  and  helps  leadership 
focus  on  understanding  and  managing  readiness  production,  cost,  and  risk. 

As  shown  in  figure  4,  the  Systems  Commands  (SYSCOMs)  are  designated  as  the  lead 
providers.  In  this  role,  the  SYSCOM  Commanders  are  responsible  for  aligning  the  total 
contingent  of  providers  consisting  of  the  SYSCOM,  SYSCOM -affiliated  Program  Executive 
Offices  (PEOs),  Office  of  Naval  Research  (ONR),  Defense  Advanced  Research  Projects  Agency 
(DARPA),  Warfare  Centers,  academe,  and  industry. 


Figure  4.  Warfare  Enterprise  Construct 


In  December  2006,  CNO  ADM  Michael  Mullen  directed  the  establishment  of  a  new 
Provider  Enterprise  to  address  future  capability  at  cost  and  to  support  the  Fleet  Readiness 
Enterprise  for  current  readiness  at  cost.  The  relationship  of  the  Provider  Enterprise  to  the 
existing  enterprises  is  shown  in  figure  5.  The  details  of  the  relationship  are  still  in  development. 
It  is  unclear  what  form  the  lead  provider  role  will  be  in  the  Provider  Enterprise.  It  is  likely  that 
these  providers  will  heavily  overlap  with  the  Warfare  Enterprise  providers. 
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Enterprise  Governance  (Post- 
Offsite)  ^ _ 


CNO  UNSECNAV 


Figure  5.  New  Enterprise  Construct  (from  reference  I 1 ) 


2.2  NAVY  WARFARE  ENTERPRISE  DEMAND  SIGNAL 
2. 2. 1  Identifying  the  Demand  Signal 

The  product  of  the  Warfare  Enterprises  is  units  ready  for  tasking  (figure  2),  which  is  the 
product  the  lead  TYCOM  brings  forward  as  the  provider  in  the  Fleet  Readiness  Enterprise 
(FRE).  This  is  consistent  with  the  TYCOM  role  of  operating,  maintaining,  training,  and 
equipping  platforms  in  support  of  Fleet  and  National  tasking.  To  meet  this  role,  the  full  demand 
signal  is  composed  of  all  the  DOTMLPF  (doctrine,  organization,  training,  materiel,  leadership 
and  education,  personnel,  and  facilities)  elements  as  they  relate  to  the  TYCOM  roles.  All  of 
these  elements  must  be  properly  aligned  to  have  an  operational  capability.  As  lead  provider  in 
the  FRE,  the  TYCOMs  have  supporting  providers,  including  training  commands  and  other  shore 
infrastructure. 

Within  the  Warfare  Enterprise,  the  “primary  Provider/Enabler  Elements  are  Manpower, 
Personnel,  Training,  and  Education  (MPT&E),  Acquisition,  Technical  Authority,  and  Logistics 
(AT&L),  Installations  Management,  Health  Care,  and  Science  and  Technology”  (reference  8). 
Many  of  these  elements  are  beyond  the  lead  SYSCOM  organizational  mission  and 
responsibilities.  As  a  result,  the  Warfare  Enterprise  provider’s  primary  emphasis  is  focused  on 
materiel — the  “M”  in  DOTMLPF.  In  a  sense,  the  provider  is  an  acquisition  enterprise. 
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Whal  does  this  mean  for  the  required  nature  of  the  demand  signal?  It  is  incumbent  on  the 
warfighter  to  properly  identify  the  demand  signal  to  the  Warfare  Enterprises.  The  demand  signal 
needs  to  be  stated  in  terms  of  required  operational  capabilities,  with  clearly  defined  and 
quantified  warfighting  metrics  (e.g.,  a  certain  probability  of  kill  against  a  particular  target  in  a 
theater  of  interest).  The  demand  signal  should  not  be  in  a  single  relative  term  such  as  maximized 
readiness  at  affordable  cost.  Maximized  readiness  at  affordable  cost  may  not  win  wars. 
Affordability  is  relative  to  an  acceptable  level  of  risk — similar  to  personal  decisions  about  how 
much  insurance  to  carry. 

Proper  warfighting  metrics  can  be  translated  into  terms  understandable  by  the  acquisition 
and  R&D  communities.  Operational  capabilities  can  be  translated  into  technical  or  system 
capabilities  that  form  the  basis  of  acquisition/R&D  projects  and  portfolios.  The  provider  can 
manage  to,  and  report  on,  the  portfolios  based  on  technical  or  system  metrics  directly  linked  to 
operational  capability  metrics. 

Although  the  burden  is  on  the  warfighter  to  provide  the  proper  demand  signal,  it  is 
sometimes  necessary  for  the  provider  to  work  with  the  warfighter  to  shape  the  demand  signal 
based  on  the  art-of-the-possible.  There  is  a  historical  precedence  for  the  warfighter  and  provider 
communities  to  co-evolve  requirements. 


2. 2. 2  Current  Demand  Signal 

The  enterprise  currently  defines  the  output  of  the  enterprise  in  terms  of  (reference  7): 
Productivity  or  Output  =  Readiness/Cost. 

Discussions  of  a  single  productivity  metric  have  lead  to  an  interpretation  that 
Readiness/Cost  =  Units  Ready  for  Tasking/Cost, 

where  the  costs  are  those  not  associated  with  operations  and  support  (O&S).  Units  become 
defined  as  ships  or  airframes  ready  for  tasking  within  the  appropriate  Warfare  Enterprise. 

This  interpretation  needs  further  examination.  For  illustration  purposes,  let's  assume  a 
fictional  ship  and  submarine-based  Warfare  Enterprise.  Assume  a  2005  scenario  where,  in  the 
best  case,  all  281  ships  are  ready  for  tasking.  Assume  a  representative  budget  of  $26B  as  the 
non-O&S  budget  associated  with  these  ships.  This  would  result  in: 

Output  =  Productivity  =  1.08e-08  ships/dollar. 

What  would  a  measure  of  nano-shi ps/do liar  mean  to  the  warfighter?  Is  the  implicit  metric 
definition  of  Units  Ready  for  Tasking/Cost  correct?  Those  advocating  a  single  metric  might 
argue  the  inverse  metric  should  be  used: 

Output  =  Cost/Readiness  =  Cost/Units  Ready  for  Tasking. 

This  would  not  detract  from  the  desire  that  cost  should  be  kept  low  for  each  unit. 
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Utilizing  the  values  from  the  previous  fictional  scenario: 

Output  =  $92,526,690/ship. 

This  is  intrinsically  more  appealing  than  nano-ships/dollar.  If  costs  were  reduced  by  an  order  of 
magnitude,  the  value  would  be  $9 ,2 5 2, 669/ship.  This  has  an  appeal  in  that  it  is  cost  sensitive. 

But  does  anyone  believe  the  Navy  is  actually  spending  an  average  on  the  order  of 
$92, 5 26, 690/existing  hull?  Composite  metrics  can  easily  send  the  wrong  message. 

This  construct  would  also  require  a  complex  set  of  assumptions.  Does  units  ready  for 
tasking  mean  only  existing  units?  What  ships  are  to  be  considered  as  those  that  are  ready  for 
tasking?  At  what  point  in  the  year?  What  costs  should  be  considered?  How  are  current  and 
future  readiness  assessed?  From  a  provider  perspective,  this  metric  does  not  capture  the 
SYSCOM  intent  to  show  how  cost  cutting  measures  are  increasing  the  Navy’s  buying  power. 

It  is  insightful  to  look  at  a  non -military  example  where  the  value  proposition  of  readiness  at 
affordable  cost  is  desired.  U-Haul*  is  in  the  business  of  providing  units  ready  for  tasking. 
U-Haul’s  value  proposition  to  the  prospective  customer  can  be  summarized  as  new,  safe, 
reliable,  and  “meets  your  needs”  equipment  (right  capability),  accessibility  to  many  assets  and 
locations  (right  time),  and  affordability  (right  cost).*  As  a  customer,  you  weigh  these  individual 
values  relative  to  your  needs.  As  a  customer,  you  need  multiple  metrics  to  determine  if  you  will 
be  able  to  achieve  your  desired  effects  with  the  available  equipment  and  at  affordable  prices. 

The  complexity  of  the  business  precludes  using  a  single  metric. 

Similarly,  personal  investment  decisions  do  not  rely  on  a  single  metric  to  understand  a 
company — its  value  proposition,  earning  potential,  and  historic  trends.  To  do  so  would 
misrepresent  the  risks  that  need  to  be  addressed  by  the  decision  maker. 

Likewise,  Warfare  Enterprise  risk  assessment  cannot  be  achieved  via  a  single  metric— the 
nature  of  the  business  does  not  allow  it.  The  bottom  line  is  that  the  current  output  construct  is  of 
limited  value  to  the  Warfare  Enterprise  decision  makers.  Senior  Navy  leadership,  similar  to  their 
corporate  peers,  needs  to  understand  and  manage  to  complex  and  multiple  business  metrics. 

The  warfighter  demand  signal  must  be  in  terms  of  capabilities  made  available  to  the 
operating  forces  based  on  correct  and  affordable  DOTMLPF.  The  resulting  enterprise  output 
metrics  should  be  in  terms  of  operational  capabilities  and  risk. 


*Value  proposition  was  derived  from  advertising  at  www.uhaul.com  . 
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2.3  WARFARE  ENTERPRISE  PROV  IDER  VALUE  PROPOSITION 


The  lead  provider  of  each  Warfare  Enterprise  is  a  lead  SYSCOM.  Each  SYSCOM  desires 
to  provide  value  to  the  Navy  in  the  manner  expressed  by  COMNAVSEA  (reference  6): 

Vision:  Be  a  responsive,  effective  and  efficient  provider  for  the  Navy  Enterprise. 

Mission:  Put  the  right  capability  in  the  hands  of  the  warfighter  at  the  right  time  at 
the  right  cost. 

In  addition,  the  SYSCOMs  are  under  pressure  to  find  ways  to  drive  down  costs  (reference  8) 
and  thus  improve  the  Navy’s  buying  power.  These  are  not  as  directly  related  as  they  may 
initially  seem,  since  the  SYSCOMs  can  only  provide  those  products  and  services  that  have  been 
appropriated  by  Congress  and  signed  into  law. 


2.3. 1  Right  Capability ,  Right  Time,  Right  Cost 

The  appropriateness  of  the  provider-delivered  products  and  services  (right  capability,  right 
time,  right  cost)  is  subject  to: 

•  OPNAV  interpretation  of  Fleet  requirements  in  the  budget  submittal 

•  Whether  the  quantities  being  procured  are  sufficient  to  provide  a  warfighting 
capability  (capacity) 

•  Congressional  determination  of  requirements  and  capacity  via  power  of  the  pen 
and  purse  (earmarks) 

•  Actual  delivery  of  the  appropriated  products  and  services  within  performance, 
cost,  and  schedule. 

The  SYSCOM  can  control  only  the  last  of  these  factors,  and  it  has  extensive  experience  in 
management  controls  to  measure  success  or  failure.  The  warfighter  and  resource  sponsor  need  to 
address  the  first  two  factors.  Although  held  accountable  by  the  warfighter,  the  SYSCOM — for 
the  reasons  given  above — cannot  control  success  in  delivering  the  right  capability  at  the  right 
time  and  the  right  cost. 

The  right  cost  is  based  on  the  perceived  value  of  the  product  or  service  by  many. 

SYSCOMs  can  establish  performance/cost  goals  and  hope  (1)  that  the  market  can  or  is  willing  to 
deliver  and/or  (2)  that  the  Government  acquisition  and  oversight  functions  can  be  delivered  to 
these  goals.  SYSCOMs  have  experience  in  measuring  progress  toward  meeting 
performance/cost  goals. 

Thus,  the  Warfare  Enterprise  needs  to  establish  warfighting  success  and  risk  metrics  for  the 
program  of  record  and  proposed  acquisi  tion  options.  These  metrics  should  clearly  show  the 
performance  based  on  war  fighting  mission  effectiveness,  warfighting  risk,  and  cost  based  on 
program  cost,  performance,  schedule,  and  risk.  This  has  an  additional  benefit — it  allows  the 
Warfare  Enterprise  to  show  the  impact  of  decisions  made  outside  of  its  control. 
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This  approach  leads  to  metrics  of  the  form: 

Output  =  Performance/Cost. 

The  provider  should  establish  metrics  that  entail  product  (system)  or  service  (process) 
measures  of  performance  that  feed  the  warfighting  measure  of  effectiveness — classic  cost/benefit 
measures. 


2.3.2  Improved  Buying  Power 

In  the  private  sector,  the  consumer  gains  buying  power  if: 

•  Corporations  elect  to  lower  selling  prices  to  increase  profits  through  higher 
volume  sales 

•  Corporations  elect  to  maintain  the  same  profit  margin  but  lower  prices  as 
manufacturing  costs  are  reduced  through  lowered  material  costs  or  improved 
manufacturing  efficiencies 

•  Corporations  reduce  selling  price  and  reduce  profit  margins — an  undesirable 
condition  for  a  profit-based  business. 


The  SYSCOMS  and  affiliated  PEOs  face  similar  conditions: 

•  Will  budget  appropriations  support  higher  volume  purchases  at  reduced  unit 
cost?  This  is  not  within  SYSCOM  control. 

•  Are  corporations  willing  and  able  to  lower  prices  any  further?  Are  they  already 
at  minimum  levels  acceptable  to  their  shareholders?  This  is  not  within 
SYSCOM  control,  but  reduced  costs  through  efficiency  improvements  can  be 
encouraged  through  incentives. 

•  Are  there  new  technologies  or  processes  that  would  reduce  manufacturing 
costs?  This  is  not  within  SYSCOM  control;  however,  the  technology 
organizations  within  the  enterprise  can  address  this  topic.  SYSCOMs  can 
encourage  this  research  through  funded  tasking  and/or  incentives. 

•  Are  there  efficiency  improvements  to  be  had  within  the  Government  acquisition 
and  oversight  process?  This  is  within  SYSCOM  control  as  constrained  by  the 
Federal  Acquisition  Register  (FAR)  and  other  regulations. 

•  Are  the  requirements  behind  the  acquisition  more  stressing  than  required  by  the 
Fleet?  Are  the  cost  drivers  behind  the  requirements  fully  understood?  This  is 
not  within  SYSCOM  control,  but  the  SYSCOM  can  work  within  the  Warfare 
Enterprise  to  validate  existing  requirements. 
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The  results  of  any  of  these  actions  lend  themselves  to  reporting  the  potential  for  improved 
“buying  power”  via: 

•  Cost  reduction  due  to  requirement  relaxations  (based  on  cost,  performance,  and 
risk  assessments). 

•  Cost  savings  due  to  Government  efficiencies  without  the  compromise  of 
inherently  Governmental  functions. 

•  Cost  savings  and  cost  avoidance  due  to  technology  infusion. 

Each  of  these  cost  reductions  positively  impacts  metrics  in  the  form  of  perfonnance/cost. 
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3.  NAVY  WARFARE  ENTERPRISE  R&D 


The  Warfare  Enterprise  provider  must  respond  to  the  readiness  demand  signals  for  the 
current  Navy,  the  future  Navy,  and  the  Navy-after- Next.  This  necessitates  that  the  provider 
engage  in  activities  that  span  the  full  spectrum  from  science  and  technology  (S&T)  to  operational 
system  support.  The  provider  meets  these  time-evolving  demands  through  parallel  S&T,  R&D, 
and  acquisition  “pipelines.”  From  an  R&D  perspective,  the  pipeline  must  contain  a  balanced 
portfolio  of  rapid-transition  applied  research  and  long-term  research. 


As  shown  in  figure  6,  these  pipelines  transition  different  output  products.  At  any  point  in 
time,  the  provider  is  providing  the  following  output  products: 


•  Acquisition  Community  -  Transition  of  purchased  material  to  the  current  Navy 

•  R&D  Community  -  Transition  of  technology  into  the  acquisition  community  for 
future  Navy  systems  or  current  Navy  system  improvements 

•  R&D  Community  -  Transition  of  basic  research  to  applied  research  for  future 
Navy  and  Navy-after-Next  systems. 


Operating 

Forces 


Time  tz) 

Current  Navy 


Future  Navy 

Product  and  Service  Transitions 


Acquisition 
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Figure  6.  R&D  and  Acquisition  Transitions 

Provider  output  metrics  need  to  be  consistent  with  these  output  products.  For  example,  the 
acquisition  community  must  be  able  to  provide  measured  system  performance  values  that  the 
warfighter  can  use  to  determine  or  measure  operational  capability  performance.  The  R&D 
community  must  measure  technology  performance  that  can  be  used  to  determine  the  expected 
value  of  systems  that  incorporate  this  technology.  The  warfighter  can  use  these  values  to 
determine  the  expected  operational  capability  if  the  technology  is  successful.  Provider  output 
metrics  need  to  account  for  the  fact  that  each  output  product  is  different  and  will  have  a  different 
effect  on  operational  capability  or  risk  reduction. 
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3.1  PARALLELS  WITH  INDUSTRY  R&D 


The  Warfare  Enterprise  is  a  collection  of  team  members  and  collaborative  processes  used  to 
align  the  people,  dollars,  and  programs  associated  with  a  warfighter  area  (e.g.,  Undersea 
Warfare)  (reference  6).  Industry  is  different  in  that  it  does  not  separate  its  “resource  sponsor" 
from  its  R&D.  The  industrial  relationship  between  functional  units  of  business  (e.g., 
manufacturing,  R&D,  marketing)  and  its  customer  has  been  extensively  studied.  This  is  the  field 
of  R&D  management. 

The  provider’s  need  for  R&D  necessitates  a  look  at  the  relationship  of  R&D  functional  units 
to  the  other  business  units  and  the  customers.  Relationship  changes  between  the  business  units 
have  evolved  into  recognized  generations  of  R&D  management.  The  generation  nomenclature 
can  be  traced  back  to  the  mid-1990s  and  can  be  defined  as  follows  (see  appendix  A  fora  more 
comprehensive  definition):* 

Generation  1.  R&D  is  an  overhead  function  insular  to  and  independent  of  the 
corporation’s  strategic  framework.  The  R&D  function  is  to  develop  technologies 
that  may  or  may  not  be  consistent  with  the  business’s  strategic  direction  and 
goals.  Technology  push  is  the  prime  emphasis. 

Generation  2.  R&D  remains  insular  but  becomes  less  independent  and  begins  to 
be  aligned  with  marketing  strategies.  Managing  technology  “pull”  begins  to 
address  particular  product  development  strategies  (needs  pull).  The  R&D 
function  is  managed  on  a  project-by-project  basis. 

Generation  3.  Senior  leadership  and  functional  managers  together  develop  the 
business  plan  and  model  based  on  the  corporate  strategy.  Strategically-balanced 
R&D  portfolios  across  the  corporation  are  developed  based  on  technology 
development  roadmaps  and  product  life-cycle  considerations. 

Generation  4.  Simultaneous  technology  push  and  pull  based  on  tight  integration 
of  leading  customers  in  the  evolution  from  concept  to  development. 

Competitiveness  is  dependent  on  the  technical  knowledge  and  capabilities  within 
but  also  outside  of  the  company.  R&D  management,  from  necessity  of  speed, 
goes  beyond  technical  product  and  process  to  include  business  and  market  models 
that  encompass  management  of  knowledge,  technology,  and  market/industry 
infrastructure. 

Generation  5.  The  management  of  information,  knowledge  and  innovation  is 
based  on  broad  horizontal  network  processes  that  are  cross-border  and  that  rely 
heavily  on  information  technology.  This  broad  network  is  operated  based  on 
strategic  partnerships  that  ensure  overall  organizational  and  systems  integration. 
Developers,  manufacturers,  and  customer  collaborate  in  defining  the  way  forward 
and  the  subsequent  research. 


This  description  of  R&D  management  generations  relies  heavily  on  those  presented  in  references  1 2  and 
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An  initial  reaction  might  be  the  desire  to  aspire  to  the  newest  generation  of  R&D 
management.  It  should  be  noted  that  the  newer  generations  apply  to  product  areas  that  are 
subject  to  extremely  rapid  technology  changes  and  intense  competition  based  on  customer 
demand  for  new  products  that  include  these  technologies,  e.g.,  consumer  information  technology 
products  such  as  cell  phones.  Industries  based  on  advanced  technology  and  hypercompetitive 
market  demand  product  areas  more  tightly  couple  R&D  to  marketable  endproducts  as  a  result  of 
significant  customer  interaction.  This  tight  coupling  is  known  by  many  names,  including  “open 
innovation”  (reference  14). 

Associated  with  advanced  R&D  management  generations  is  a  recognized  danger  that  must 
be  managed.  The  tight  coupling  of  R&D  in  a  competitive  climate  has  led  to  an  emphasis  on 
rapid  transition  (and,  thus,  rapid  applied  research)  at  the  expense  of  longer-term  technology 
research.  This  course  leads  to  a  long-term  exposure  to  competitors  if  decision  makers  do  not 
maintain  a  proper  balance  of  short-  and  long-term  research  in  their  R&D  portfolio.*  From  a 
military  perspective,  this  competition  might  result  in  losing  technological  advantage  to  future 
adversaries,  t 

Not  every  product  area  needs  to  be  generation  5.  It  is  probably  safe  to  assume  that  the 
Warfare  Enterprise  primarily  operates  at  the  generation  3  level  across  its  product  areas,  with  tails 
to  generations  2  and  4.  As  lead  provider,  the  SYSCOM  must  take  the  lead  in  establishing  the 
appropriate  level  of  R&D  management  within  the  Warfare  Enterprise  business  model.  More 
advanced  R&D  management  generation  requires  greater  adherence  to  the  principles  of  open 
innovation  and  establishes  the  required  nature  of  the  relationships  and  concept  of  operation 
between  the  warfighter,  resource  sponsor,  and  provider. 


3,2  NAVY  WARFARE  ENTERPRISE  R&D  OUTPUT  METRICS 

5.2. 1  Measuring  R&D  and  the  Impact  of  the  R&D  Product 

Within  the  military  context,  the  ultimate  purpose  of  R&D  is  to  broaden  and  deepen  military 
warfighting  capabilities.  At  the  next  lower  level,  the  purpose  of  U.S.  federally  funded  R&D  is 
to: 


•  Gain  knowledge  or  understanding  of  scientific  principles  with  presumed  broad 
application  to  needs  and  requirements 

•  Determine  the  means  by  which  specific  needs  and  requirements  may  be  met 
through  new  technologies  or  knowledge 

•  Apply  knowledge  to  the  creation  of  new  and  improved  processes,  products,  and 
services  that  meet  specific  needs  or  requirements.* 


•Many  references  exist;  one  example  is  reference  15. 

tMuch  has  been  written  over  the  last  few  years  on  the  potential  erosion  of  U.S.  technological  advantage.  For 
examples,  see  references  16  and  17. 

fThese  are  an  adaptation  of  definitions  for  basic  research,  applied  research,  and  development  from  reference  18. 
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To  develop  R&D  metrics,  it  is  important  to  understand  the  actual  R&D  output  product. 
Roussel  et  al.  (reference  19)  state: 

R&D  produces  one  product  only  -  knowledge.  True  it  is  knowledge  with  a 
purpose,  but  it  is  still  just  knowledge.  R&D  does  not  produce  a  physical  product 
for  sale  or  an  operating  process.  It  does  not  produce  a  new  business.  Nor  does 
it  produce  quality.  However,  R&D  does  produce  the  know-how  al  the 
foundation  of  all  these  other  results. 

In  almost  all  companies  across  the  world,  the  know-how  developed  by  R&D 
must  be  translated  by  management  action  into  products,  processes,  cost 
reductions,  quality  improvements,  conformance  with  environmental  regulations, 
support  of  product  claims,  and  other  objectives.  Rarely  is  this  know-how 
created  strictly  for  sale. 

In  short,  R&D  seeks  out  the  location  of  the  treasure,  but  senior  business 
management  holds  the  key  to  it.  Only  management  can  mobilize  all  of  the 
resources  necessary  to  transform  an  R&D  result  -  knowledge  -  into  a 
commercially  useful  result.  Only  management  can  provide  support  from 
marketing,  manufacturing,  or  capital.  Only  management  can  enforce  the 
company’s  vision  and  strategies  and  involve  all  functions  -  including  R&D  in 
their  successful  implementations. 

Relative  to  the  public  sector,  Kostoff  (reference  2)  states: 

Public-sector  S&T  sponsors  have  two  major  responsibilities:  a)  to  sponsor  high 
quality  S&T  that  has  high  potential  for  eventually  being  used  to  improve 
systems  and  operations  of  the  sponsor’s  stakeholders/customers  for  national 
benefit,  and  b)  to  make  the  downstream  developcrs/acqui  sit  ioners  of  these  final 
products  aware  of  global  S&T  being  performed  that  could  impact  their 
downstream  development  and  acquisition.  These  S&T  sponsors  have  little 
control  over  the  fate  of  their  sponsored  S&T  after  the  S&T  is  completed,  and 
especially  after  the  S&T  transitions  to  other  organizations  for  further 
downstream  development  and  acquisition.  Some  of  the  many  external  factors 
that  determine  the  eventual  fate  of  S&T  other  than  technical  quality  include 
geopolitical,  local  political,  economic,  financial,  legal,  environmental,  cultural, 
etc.  The  only  control  the  S&T  sponsors  can  actually  exert  over  potential 
applications  is  lo  produce  a  high  quality  product  that  has  positive 
Iransitionability  characteristics  (e.g,,  affordable,  maintainable,  reliable, 
addresses  stakeholder  and  customer  need,  high  technical  quality,  etc). 

Succinctly.  S&T  sponsors  control  outputs,  not  outcomes. 

Yet,  present  metrics  systems  for  evaluating  public  sector  S&T  sponsors  do  not 
address  the  reality  of  the  two  responsibilities  described  above.  Public  sector 
S&T  sponsors  are  held  accountable  for  both  outputs  and  outcomes.  Many 
public  sector  S&T  sponsor  evaluations  contain  metrics  that  address  downstream 
outcomes.  Public  sector  S&T  sponsors  are  held  accountable,  to  some  degree, 
for  S&T  products  that  do  not  transition  for  further  development,  or  that  do  not 
eventually  result  in  envisioned  outcomes.  This  is  an  example  where  the 
appropriateness  of  the  metric  is  perhaps  more  important  than  its  measurement 

capability. 
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The  parallel  to  the  Warfare  Enterprise  is  in  the  method  by  which  “technology”  actually 
transitions  to  the  Fleet.  In  reality,  technology  transitions  only  when  it  is  incorporated  into  a 
system  purchased  for  the  warfighter.  Due  to  performance-based  procurement  regulations,  the 
vendor  providing  the  material  has  the  latitude  to  determine  which  technologies  are  utilized.  The 
vendor  may  or  may  not  elect  to  use  technologies  developed  by  Navy  S&T  or  R&D.  Technology 
transition  agreements  notwithstanding,  there  is  no  guarantee  that  a  particular  technology  will  be 
manifested  in  acquisition  system  designs. 

The  R&D  conducted  by  the  Navy  serves  a  number  of  purposes.  It  explores  technology 
possibilities  not  necessarily  deemed  to  have  a  sufficient  return-on-investment  (ROI)  for 
commercial  purposes.  It  allows  exploration  of  proof  of  concept,  which  provides  the  warfighter 
an  understanding  of  the  realm  of  the  possible.  It  provides  the  acquisition  agent  the  knowledge 
from  which  to  develop  specifications,  make  smart  buyer  decisions,  and  monitor  the  progress  of 
the  vendor.  Representative  of  Kostoff  s  observations.  Navy  R&D's  only  guaranteed  product  is 
knowledge,  and  it  is  the  purpose  of  the  collective  Warfare  Enterprise  (provider,  warfighter,  and 
resource  sponsor)  to  transition  this  knowledge  into  processes,  products,  and  services.  Within  the 
enterprise,  transition  is  a  collective  effort  requiring  collective  responsibility  and  accountability. 

Business  management  literature  discusses  the  value  of  R&D  and  the  success  of  R&D  as 
separate  items.  The  value  is  related  to  the  effect  of  the  R&D  application  on  the  end-user.  Value 
metrics  are  measures  of  effectiveness  (MOEs).  Success  is  related  to  the  extent  to  which  the 
specific  R&D  project  objectives  are  met.  Success  metrics  are  measures  of  performance 
(MOPs).* 


3.2.2  Value  MOEs 

Many  successful  generation  3  and  4  organizations  manage  R&D  through  the  management  of 
an  R&D  portfolio  that  is  linked  to  corporate  goals.  The  projects  within  the  portfolio  are  selected 
based  on  the  expected  value  of  the  new  technology  in  meeting  needs  or  requirements. 

Researchers  such  as  Geisler  are  careful  to  differentiate  expected  value  between  the  private  and 
public  sectors.  Expected  value  in  the  private  sector  is  based  on  perceived  market  share,  profits, 
and  return  on  equity  (ROE)  or  return  on  assets  (ROA).  Expected  value  in  the  public  sector  is 
based  on  contributions  to  society  and  the  economy  (reference  1).  In  the  public  sector,  societal 
benefits  (e.g.,  national  defense  and  environmental  safety)  can  have  a  greater  weight  than 
maximizing  financial  returns. 

There  are  numerous  methods  for  assessing  the  expected  value  of  R&D  portfolios.  Many  are 
based  on  the  population  of  some  form  of  potential  reward  versus  risk  chart  (reference  19).  One 
variation  is  the  capability-performance  potential  versus  transition  probability  chart  developed  as 
part  of  the  RAND  PorlMan  Decision  Framework  (figure  7),  which  is  taken  from  reference  20. 
(Reference  20  reports  on  the  RAND  adaptation  of  its  framework  for  ONR.) 


Kostoff  (reference  2)  uses  the  term  “outcome"  to  describe  the  value  metric  and  “output”  to  describe  the 
success  metric. 
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Figure  7.  RAND  PortMan  Decision  Framework  (from  Reference  IS) 

The  RAND  framework  allows  the  placement  of  R&D  projects  in  a  common  context,  which 
permits  evaluation  of  the  individual  and  collective  expected  value.  The  R&D  portfolio  manager 
(or  potentially  a  Warfare  Enterprise  Board)  can  readily  see  the  attributes  and  emphasis  of  the 
portfolio.  Is  it  a  collection  of  high  transition  probability,  incremental  improvements?  Is  it  a 
balanced  mix  of  high  transition  probability,  incremental  improvements,  and  higher  risk  but 
paradigm-changing  improvements?  Does  the  R&D  have  the  potential  of  meeting  the 
warfighter's  expectations? 

There  are  many  techniques  and  criteria  that  could  be  used  to  determine  how  and  where  a 
project  should  be  mapped  on  a  reward  versus  risk  chart.  The  referenced  RAND  work  provides 
one  technique  and,  as  an  example,  the  RAND  criteria  are  provided  in  appendix  B. 


3.2.3  Success  MOPs 

The  value  metrics  are  based  on  an  input  of  success  metrics.  Success  metrics  are  a  measure 
of  how  well  the  project  is  performing  to  its  R&D  objectives.  The  R&D  objective  might  be  an 
evaluation  of  the  level  of  performance  achievable  from  various  technologies  or  algorithms,  or  it 
might  be  a  proof-of-concept  evaluation  or  a  technology  demonstration  for  a  system.  Success 
metrics  are  not  a  determination  of  the  ultimate  success  of  the  R&D  output  to  the  end-user.  This 
is  a  difference  akin  to  the  technical  evaluation  (TECHEVAL)  and  operational  evaluation 
(OPEVAL)  construct  used  by  the  DoD.  Whether  a  system  employing  the  R&D  ultimately  has 
warfighter-sanctioned  operational  value  is  not  an  R&D  success  MOP  by  this  construct. 
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Table  I  is  an  example  of  individual  R&D  project  evaluation  criteria.  The  contents  of  the 
table  are  an  adaptation  from  Roussel  et  al.  (reference  19).  The  criteria  address  the  assumed 
benefits  of  a  successful  project,  the  probabilities  of  achieving  this  success,  project  management 
cost  and  schedule,  etc.  These  items  are  standard  criteria  familiar  to  project  managers. 

Project  attractiveness  elements  are  the  project  linkage  to  the  warfighter  demand  signal.  If 
the  warfighter  demand  signal  is  properly  stated  in  quantified  operational  capability  terms,  the 
R&D  community  can  work  with  the  warfighter  to  translate  the  operational  capabilities  into 
technical  or  system  capability  terms  recognizable  by  scientists  and  engineers.  Subsequently,  the 
R&D  project  objective  defines  the  relationship  of  the  project  to  the  technical  or  system 
capabilities  requirements.  Done  properly,  a  project  works  with  the  operational  analysis 
community  to  link  the  project  results  with  expected  operational  capability  improvements. 


Table  l.  Potential  R&D  Project  Evaluation  Criteria* 


Evaluation  Criteria 

Measure 

Project  Attractiveness 

Fit  with  enterprise  strategy 

A  judgment  ranging  from  excellent  to  poor 

Inventive  merit  and  strategic 
importance  to  the  enterprise 

The  potential  power  of  the  sought-aficr  result  to: 

•  Improve  U.S.  technological  advantage 

•  Apply  to  more  than  one  warfare  area 

•  Provide  foundation  for  the  future  Navy 

A  judgment  from  high  to  low 

Durability  of  U.S.  technological 
competitive  advantage 

Years  -  If  the  R&D  result  can  be  quickly  and  easily  initiated  by 
adversaries,  the  project  is  less  attractive  than  one  that  provides  a 
protected,  long-term  advantage 

Reward 

Usually  performance/cost  benefit,  but  sometimes  “necessity 
work”  (e.g.,  regulatory)  or  building  a  knowledge  base  that 
becomes  the  foundation  for  transitions  (next  level  of  R&D  or 
acquisition) 

Competitive  impact  of 
technologies 

Base,  key,  pacing,  embryonic  in  nature  (technology  readiness 
level) 

Project  Uncertainty 

Probability  of  technical  success 

Probability  that  the  objective  will  be  achieved  as  desired 

Probability  of  end-user  success 

Probability  of  end-user  success  if  the  project  is  technically 
achieved  (i.e.,  meets  operational  requirements) 

Probability  of  overall  success 

Probability  -  The  product  of  the  technical  and  end-user 
probabilities 

Project  Exposure 

R&D  costs  to  completion  or  key 
decision  point 

Dollars 

Timc-to-completion  or  key 
decision  point 

Time 

Cost  required  to  exploit  technical 
success  to  ready  for  transition 

Dollars 

*  Adapted  from  Roussel  el  al.  (reference  19). 
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3.2.4  Warfare  Enterprise  R&D  Metric  Hierarchy 

The  result  of  the  above  discussion  is  that  there  is  a  hierarchy  of  R&D  metrics  for  the 
Warfare  Enterprise,  as  shown  in  table  2,  that  needs  to  be  developed  and  managed  at  the  Warfare 
Enterprise  level. 


Table  2.  Hierarchy  of  Warfare  Enterprise  R&D  Metrics 


Level  of 
Evaluation 

Nature  of 
Metrics 

Current  Navy 

Future  Navy 

Navy-After-Next 

Warfare 

Enterprise 

Enterprise 
output  linked 
to  demand 
signal 

Improvement  in 
warfighting  MOE 
measured/cxpecled 
value) 

Improvement  in 
warfighting  MOE 
(expected  value) 

Improvement  in 
warfighting  MOE 
(expected  value) 

Lead  Provider 

Enterprise 
provider 
output  & 
execution 

Assessment  of 
engineering 
development 
portfolio 

Assessment  of 
advanced 
development 
portfolio 

Assessment  of 
technology  (S&T) 
portfolio 

Provider 

Project 
output  & 
execution 

•  Cost 

•  Performance 

•  Schedule 

•  Risk 

■  Cost 

•  Performance 

•  Schedule 

•  Risk 

•  Cost 

•  Performance 

•  Schedule 

•  Risk 

3.3  PRODUCTIVITY  AND  R&D  ORGANIZATIONAL  METRICS 
3.3.  /  Productivity ,  Efficiency ,  Effectiveness 

The  Warfare  Enterprises  have  spent  a  considerable  amount  of  time  trying  to  develop  a  single 
output  metric  based  on  the  concept  of  productivity.  This  section  addresses  the  concepts  of 
productivity,  efficiency,  and  effectiveness  and  the  accepted  definitions  and  metrics  for  these 
terms. 

First,  some  definitions  (from  reference  2 1 ): 

•  Productivity  is  the  creation  of  goods  and  services  to  produce  wealth  or  value. 

•  Efficiency  is  the  effective  operation  of  a  business  or  performance  of  a  business 
task  with  minimum  wasted  effort  (time,  energy,  money). 

•  Effectiveness  is  production  of  the  desired  results. 

In  short,  productivity  is  about  the  use  of  resources,  efficiency  is  doing  things  right,  and 
effectiveness  is  doing  the  right  things  (reference  22). 
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Warfare  Enterprise  provider  productivity  is  based  on  delivering  value  to  the  warfighter 
through  the  enterprise’s  products  and  services.  The  provider  value  comes  from  being  effective  in 
delivering  innovation,  transitioning  products  and  services,  and  stewarding  capabilities,  while 
maintaining  cost  effectiveness  through  business  efficiencies. 

Productivity  also  has  a  more  formal  economic  definition.  Sharpe  (reference  23)  writes: 

Productivity  is  the  relationship  between  the  output  of  goods  and  services  and  the 
inputs  of  resources,  human  and  non-human,  used  in  the  production  process,  with 
the  relationship  usually  expressed  in  ratio  form. 

The  ratio  form  of  the  definition  is:  Productivity  =  Output/Input.  It  is  obvious  why  the  Warfare 
Enterprise  selected  a  productivity  ratio  of  production  of  readiness/cost. 

The  concept  is  good  but  further  exploration  of  productivity  as  a  metric  is  required.  In 
economic  terms,  productivity  is  a  measure  of  output  quantities  to  input  quantities.  Productivity 
plus  pricing  factors  determine  profitability  (reference  22).  All  else  being  equal,  increases  in 
productivity  result  in  increased  profitability,  which  increases  economic  growth  and,  thus,  the 
standard  of  living. 

An  example  of  a  productivity  metric  is  the  gross  domestic  product  (GDP)  per  capita,  which 
is  a  measure  of  the  monetary  value  of  a  country’s  goods  and  services  produced  in  a  year 
(output)/number  of  working  people  (input).  For  the  members  of  the  International  Monetary  Fund 
in  2005,  this  metric  results  in  a  range  from  $69,800/capita  for  Luxembourg  to  $596/capita  for 
Malawi  (reference  24).  There  is  no  right  absolute  value  to  achieve.  The  measure  is  used  for 
comparison  of  one  country  relative  to  another.  The  relative  comparisons  are  frequently  used  to 
establish  national  economic  policies  geared  toward  improving  or  maintaining  certain  standards 
of  living. 

There  are  comparable  business  metrics  that  take  the  form  of  output/input.  Some  use  billable 
hours/ full  time  employees  or  revenue /full  time  employees.  These  are  not  applicable  to  the 
Warfare  Enterprise  since  its  mission  is  not  profit-oriented  and  not  to  maximize  revenue.  Others 
use  tasks  completed/full  time  employee  or  quantities  met/full  time  employees.  These  are 
applicable  metrics  where  repetitive  tasks  or  “piece  part"  work  is  the  order  of  business.  This  may 
apply  to  Warfare  Enterprise  depot  work  and  other  industrial  operations,  but  not  to  the 
enterprise’s  R&D  work. 

Looking  at  efficiency  metrics,  it  is  useful  to  examine  non-profit  organizations.  If  you  want 
to  make  a  choice  of  donating  to  one  of  three  charities  that  perform  the  same  mission,  you  would 
presumably  choose  the  charity  that  maximizes  the  use  of  its  funds  for  charitable  services.  This  is 
the  efficiency  of  the  charity  (presumed  by  many  to  also  be  a  measure  of  its  effectiveness). 
Charities  are  rated  according  to  a  charity  efficiency  rating ,  where 

Output/input  =  funds  toward  mission/funds  toward  administrative  functions. 
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Shown  below  are  some  charity  efficiency  ratings  for  2004  (reference  25): 


•  Public  Broadcast  Service 

•  American  National  Red  Cross 

•  United  Way 

•  Habitat  for  Humanity 

•  National  Public  Radio 

•  American  Heart  Association 

•  Museum  of  Fine  Art,  Boston 


96% 

91% 

88% 

82% 

78% 

75% 

64% 


The  differences  in  these  ratings  indicate  that  it  is  a  personal  decision,  probably  subjective,  as 
to  what  constitutes  perceived  value  from  a  donation. 

It  is  important  to  note  that  different  charity  types  have  different  demands  and  levels  of 
administrative  expense.  The  following  is  an  example  of  median  charity  efficiency  ratings  for 
two  types  of  charities: 

•  Food  Banks  median  94% 

•  Art  museums  median  71%. 

Thus,  it  is  important  to  understand  what  relative  efficiency  is  reasonable  for  a  particluar  type 
of  non-profit  business. 

3.3.2  Relationship  to  Warfare  Enterprise  Providers 

The  efficiencies  (direct  cost  associated  with  delivering  products  and  services/total  costs)  of 
different  provider  business  units  should  also  be  different  and  consistent  with  the  type  of  business 
function.  Repetitive,  task-oriented  units  such  as  depots  and  shipyards  strive  for  very  high 
efficiency  ratings.  R&D  is  non- repetitive  by  nature,  so  its  efficiencies  should  be  lower. 

Intuitively,  the  efficiency  of  industry  R&D  should  be  higher  than  university  R&D  (see 
figure  8).  Industry  desires  to  transition  its  entire  R&D  and  maximize  profit.  In  general,  the 
prime  motivation  for  university  R&D  is  knowledge — not  profit.  In  the  DoD.  the  efficiency  of 
the  Warfare  Centers  and  University-Affiliated  Research  Centers  (UARCs)  should  conceivably  fit 
somewhere  between  that  of  industry  and  universities,  since  transition  is  important  but  so  is  the 
conduct  of  necessary  research  not  deemed  profitable  by  industry  or  of  interest  to  universities. 

Similar  to  charities,  it  is  important  to  recognize  where  a  non-profit  R&D  organization  should 
be  positioned.  If  the  Warfare  Center/UARC  efficiencies  get  too  high,  it  is  probably  indicative  of 
( 1)  cost  cuttings  that  have  a  long-term  impact  on  R&D  health,  (2)  a  concentration  on  low-risk 
R&D,  and/or  (3)  other  factors  that  jeopardize  the  stewardship  of  technical  capabilities  necessary 
to  maintain  technical  authority.  If  Warfare  Center/UARC  efficiencies  are  too  low,  it  is  probably 
indicative  of  ( 1 )  a  concentration  on  high-risk,  long-term  R&D,  (2)  a  lack  of  propensity  to 
transition,  and/or  (3)  other  factors  that  jeopardize  the  mission  of  providing  value  to  the 
warfighter.  The  Warfare  Centers/UARCs  need  to  manage  their  efficiencies  within  a  hand.  More 
efficiency  is  not  always  better  given  the  organizational  mission  and  value  proposition. 


22 


High 


c 

0 


it 

m 


Low 


Defense  Ind.  WC/UARC  Universities 


Figure  8.  Relative  Organizational  R&D  Efficiencies 


Very  preliminary  analysis  indicates  that  recent  “cost-saving”  initiatives  at  the  Warfare 
Centers,  as  illustrated  in  figure  9,  may  be  too  extreme.  This  figure  shows  efficiency  ratings  for 
five  of  the  top  defense  industries,  the  NAVSEA  Warfare  Centers,  and  universities. 

The  university  rating  is  the  U.S.  national  average  (reference  26).  Understandably,  there  are 
high  costs  associated  with  university  research  facilities  and  libraries,  as  well  as  other 
administrative  costs. 

The  defense  industry  rating  is  based  on  a  “back-of-the-envelope”  calculation  for  direct 
costs/indirect  costs  from  corporate  annual  reports.*  One  could  argue  that  the  decreasing 
efficiencies  for  the  defense  industries  listed  in  figure  9  are  correlated  to  the  increasing  percentage 
of  R&D  performed  by  the  individual  corporations.  A  rigorous  analysis  would  be  required  to 
determine  an  exact  average  for  defense  industries. 

The  Warfare  Center  rating  is  for  the  NAVSEA  Warfare  Centers. 

The  preliminary  data  suggest  that,  given  the  R&D  nature  of  its  business,  the  NAVSEA 
Warfare  Centers  have  gone  too  far  with  cost  cutting.  Short-term  cost  cutting  can  be  managed, 
but  long-term  cost  cutting  in  the  name  of  cost  savings  can  be  detrimental. 

From  an  organizational  perspective,  productivity  should  not  be  confused  with  efficiencies 
that  result  in  cost  savings  and  cost  avoidance.  Warfare  Enterprise  providers,  in  particular  the 
R&D  elements,  must  carefully  avoid  short-term  cost  cuttings — deemed  cost  “savings” — that 
might  jeopardize  the  provider’s  long-term  ability  to  maintain  technical  authority. 


•These  include  the  Anleon  International  2004  annual  report.  General  Dynamics  2005  annual  report,  Boeing  2005 
annual  report,  Lockheed  Martin  2005  annual  report,  and  Teledyne  Technologies  2005  annual  report. 
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Figure  9.  Preliminary  Efficiency  Data 
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4.  CONCLUSIONS 


This  report  has  discussed  the  formulation  of  provider  output  metrics  for  Navy  Warfare 
Enterprises  and  associated  R&D-focused  organizations.  The  Navy  Enterprise  construct  requires 
that  upper-tier  enterprises  delineate  quantified  warfighter  requirements  (metrics  with  identified 
units  of  measure)  that  constitute  the  “demand  signal”  to  the  provider.  Properly  constructed 
demand  signal  metrics  can  be  used  to  develop  meaningful  Warfare  Enterprise  and  provider 
output  metrics.  The  existing  concept  of  maximized  readiness  at  affordable  cost  is  a  universal 
good,  but  a  single  output  metric  for  the  readiness/cost  ratio  cannot  be  developed  in  any 
meaningful  form.  Units  of  measure  cannot  be  assigned  to  the  ratio,  nor  would  the  resulting  value 
provide  any  meaning  to  a  decision  maker.  The  output  metrics  need  to  be  based  on  products  and 
services  and  should  not  be  a  measure  of  activity  (i.e.,  the  process).  The  provider  can  use  time- 
honored  portfolio  management  techniques  to  provide  measured  and/or  expected  value 
assessments  relative  to  the  warfighter  demand  signal  metrics. 

Also  discussed  was  organizational  R&D  performance  output  metrics.  Productivity  should 
not  be  confused  with  efficiencies  that  result  in  cost  savings  and  cost  avoidance.  Warfare 
Enterprise  providers,  in  particular  the  R&D  elements,  must  carefully  avoid  short-term  cost 
cuttings — deemed  cost  “savings” — that  might  jeopardize  their  long-term  ability  to  maintain 
technical  authority. 
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APPENDIX  A 

GENERATIONS  OF  R&D  MANAGEMENT 


GENERATION  I 

R&D  is  an  overhead  function  insular  to  and  independent  of  the  corporation's  strategic 
framework.  The  R&D  function  is  to  develop  technologies  that  may  or  may  not  be  consistent 
with  the  business’s  strategic  direction  and  goals.  Technology  push  is  the  prime  emphasis.  Leave 
the  R&D  laboratory  alone  and  good  things  will  eventually  happen.  This  will  generate  new 
products  for  the  business.  Rely  on  “legacy”  products  until  the  breakthroughs  occur. 


GENERATION  2 

R&D  remains  insular  but  becomes  less  independent  and  begins  to  be  aligned  to  marketing 
strategies.  Managing  technology  “pull”  begins  to  address  particular  product  development 
strategies  (needs  pull).  The  R&D  function  is  managed  on  a  project-by-project  basis.  An 
integrated  corporate  view  is  difficult  to  obtain  and  corporate  priorities  across  projects  are  hard  to 
establish;  in  fact,  it  is  hard  to  establish  priorities  within  business  sectors.  Marketing  develops 
new  ideas  and  R&D  delivers  the  product.  Business  management  literature  refers  to  this  as  being 
a  past  generation;  many  companies  still  practice  this  generation. 


GENERATION  3 

This  is  the  first  generation  that  is  based  on  corporate  strategy.  Senior  leadership  and 
functional  managers  together  develop  the  business  plan  and  model  based  on  the  corporate 
strategy.  Strategically  balanced  R&D  portfolios  across  the  corporation  are  developed  based  on 
technology  development  roadmaps  and  product  life-cycle  considerations.  R&D  develops  new 
ideas,  and  market  feedback  refines  the  product.  This  partnership  of  general  and  R&D  managers 
ensures  that  R&D  provides  the  product  needed  to  meet  the  corporate  strategy.  Most  advanced 
R&D  corporations  manage  to  the  generation  3  model.  This  is  also  referred  to  as  the  coupling  of 
technology  push  and  pull. 


GENERATION  4 

Generation  4  begins  the  adaptation  to  the  pace  of  new  technology  development  cycles  and 
the  competitive  pace  of  new  products  that  exploit  these  cycles.  There  is  simultaneous 
technology  push  and  pull  based  on  leading  customers  tightly  integrated  in  the  evolution  from 
concept  to  development.  Competitiveness  is  dependent  on  the  technical  knowledge  and 
capabilities  within  but  also  outside  of  the  company.  Knowledge  base  (repository  for,  integration 
of,  and  body  of)  is  the  critical  component  of  the  company.  This  broad  knowledge  base  allows 
for  simultaneous  and  parallel  innovation  versus  the  serial  innovation  that  is  the  basis  of  the  first 
three  generations.  Discontinuous  (disruptive)  innovation  is  desired.  R&D  management,  from 
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necessity  of  speed,  goes  beyond  technical  product  and  process  to  include  business  and  market 
models  that  encompass  management  of  knowledge,  technology,  and  market/industry 
infrastructure.  Dynamic,  proactive,  and  energetic  management  and  business  models  are  key 
aspects  of  this  generation.  Generation  4  management  is  not  widely  found. 


GENERATION  5 

Generation  5  takes  Generation  4  to  another  level  of  emphasis  on  flexibility  and  speed  of 
development.  The  management  of  information,  knowledge,  and  innovation  is  based  on  broad 
horizontal  network  processes  that  are  cross-border  and  that  rely  heavily  on  information 
technology.  This  broad  network  is  operated  based  on  strategic  partnerships  that  ensure  overall 
organizational  and  systems  integration.  Knowledge  base  and  collaboration  are  key  attributes. 
Developers,  manufacturers,  and  customer  collaborate  in  definition  of  the  way  forward  and  the 
subsequent  research.  Generation  5  management  is  still  emerging. 
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APPENDIX  B 

EXAMPLE  OF  R&D  PORTFOLIO  ASSESSMENT  CRITERIA 


Note:  The  material  in  this  appendix  is  taken  from  appendix  C  of 
reference  18. 
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APFEMC4X  C 


Value  Descriptions  and  Scales 


Value  of  $&T  projects  will  be  based  on  three  factors: 

Value  =  Capability  *  Performance  Potential  *  Transition  Probability 
Each  factor  will  be  estimated  based  on  answers  to  the  following  questions: 


Capability 

The  capability  that  the  project  is  aimed  at  achieving  will  be  evaluated  based  on  its 
importance,  at  a  specified  level  ol  performance,  to  a  sped  tic  war  lighting  scenario,  as 
well  as  the  extent  to  which  it  influences  ocher  important  scenarios. 

Assuming  t he  project  Is  fully  successful,  the  resulting  capability  would  be; 

□  Critical  to  success  in  the  scenario 

□  A  mafot  tactor  for  success  in  the  scenario 

□  Helpful  to  success  in  the  s:enaria 

□  Not  relevant  to  or  possibly  detrimental  to  success  in  the  scenario 

How  would  you  assess  the  applicability  of  this  resulting  capability  across  Important 
scenarios? 

□  Pervasive  across  many  scenarios 

□  Useful  in  a  number  of  diflerent  scenarios 

□  Applicable  to  a  very  limited  number  ot  scenarios  similar  to  this  one 

Perto nuance  Potential 

The  performance  potential  will  be  evaluated  based  on  the  extern  to  which  the  project 
may  provide  performance  consistent  with  achieving  the  required  capability. 
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Assuming  the  project  Is  fully  successful,  the  performance  needed  to  achieve  the 
required  capability  for  the  scenario  would  be: 

□  Fully  achieved 

□  Part  k  Hy  a  ch  icved 

□  Hardly  a  chi  cvcd  at  all 

Assuming  that  the  project  Is  fully  successful,  the  performance  descrlhecl  above 
would  be  achieved  under: 

□  All  relevant  scenario  conditions 

□  Most  re  Want  scenario  conditions 

□  Some  relevant  scenario  conditions 

□  A  limited  number  or  none  of  the  relevant  scenario  conditions 


Transition  Probability 

The  transition  probability  will  be  evaluated  based  on  die  quality  of  the  transition  plan 
and  the  difficulty  of  remaining  technical  and  fielding  problems. 

The  project  and  project  team  Is  presently  characterized  as: 

□  No  remaining  technical  problems;  experience  fielding  similar  technology 

□  Remaining  technical  problems  experience  fieldt ng  sim ilar  technology 

□  No  remaining  technical  problems;  no  experience  fielding  similar  technology 

□  Remaining  technical  problems:  no  experience  fielding  similar  technology 

The  transition  plan  for  this  project  Is: 

□  Well  conceived  and  appears  to  be  i mpi cm en  table 

□  Has  some  problems  with  cost*  schedule,  or  other  fielding  burdens 

□  Has  major  problems  with  cost,  schedule,  or  other  fielding  burdens 

□  Is  severely  flawed  or  nonexistent 


Capability  Scale 


PervaHve 

Number  of  Different 

Limited  Number 

Critical 

5 

4 

3 

Major  Factor 

4,5 

3.5 

2.5 

Helpful 

4 

3 

2 

Not  Relevant 

3 

2,5 

1 
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Performance  Potential  Scale 


Fully 

Partially 

Hardly  At  aji 

All 

5 

4 

2.5 

Most 

4 

2 

2 

Some 

3 

23 

1 

Limited  or  None 

2 

1 

0 

Transition  Probability  Scale 

No  Technical  Problems 
Experience  Fielding 

No  Technical  Problems 
No  Experience  Fielding 
or  Technical  Problems 
Experience  Fielding 

Technical  Problems  No 
Experience  Fielding 

implement*  He 

S 

4 

2 

Some  Problems 

4 

2 

2 

Major  Problems 

2 

2 

1 

Severely  Flawed 

2 

1 

0 
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